feat(core): add deepAssert evidence-chain assertions - #3099
Open
xzh6656 wants to merge 1 commit into
Open
Conversation
Judge assertions from action before/after screenshots and recent task history instead of only the last frame. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
原先的
aiAssert只根据最后一张截图判断断言是否成立。静态页面检查够用,但一旦断言依赖「刚刚发生过的动作」,这套证据就不对了:toast 已经消失、歌曲只在动作窗口里切过、提交后表单看起来没变、按钮状态只在before-calling和after-calling之间翻转。本 PR 增加 deepAssert 证据链断言。默认开启,并兼容旧的三参
aiAssert(assertion, message, options)。deepAssert 做的事:
before-calling和编号后的after-calling-N(由AfterActPictures、Interval控制)。AssertionContextBoundary(lastAssert/session)、BeforeExecutions、BeforeTasks划窗口。MaxPictures。MaxPictures = 0直接失败。没有历史图时回退当前截图,并标记assertionEvidenceFallback: 'currentScreenshot'。参考图N / name。{ deepAssert: false }仍走原来的「只看最后一帧」行为。要解决的问题
现在的断言是一次 Insight extract,只看最新 UIContext。实际会踩三个坑:
deepAssert 把断言改成对有边界的动作证据链做复核,而不是对单张截图做一次性问答。
实现路径
改动接到现有执行、dump、报告链路上,没有另起一套 runner。
1. 公开选项
AiActOptions:AfterActPictures(默认1),Interval(默认50ms)。AgentAssertOpt/InsightAPI.aiAssert:deepAssert(默认true),AssertionContextBoundary(默认'lastAssert'),BeforeExecutions(默认1),BeforeTasks(默认1),MaxPictures(默认2)。aiAssert支持二参「断言 + 选项」,也兼容旧三参。@midscene/test工作流节点透传同样字段。2. 录制动作证据
TaskRunner只在 Action Space 任务上挂证据帧:before-callingafter-calling-1..N,帧之间隔Interval非动作任务的最后一个 task 仍用原来的单张
after-calling,Planning / Insight 的报告表现不变。3. 为断言选历史
TaskExecutor通过getAssertionExecutions: () => this.dump.executions拿到会话历史。在
createTypeQueryExecution('Assert')里:lastAssert:只取上一次断言之后的 executionsession:取当前会话里最近的 executionBeforeTasks个任务MaxPictures张不重复的 Action Space 图,最新在前4. 模型输入
AiExtractElementInfo把具名证据图当作主要视觉输入。有证据图时不再附当前截图。上下文里写明断言目标、所选 execution、Task N引用,以及图名。没有历史图时回退当前截图,并写
assertionEvidenceFallback: 'currentScreenshot'。5. thought 与报告
composeAssertThought()把模型回复整理成四段中文,并写入逐条 task 引用。报告 / 回放通过buildDeepAssertScreenshots()/deepAssertEvidence()读assertionEvidenceImages,展示时反转为最旧在前,标签参考图N / name。默认值与关闭方式
deepAsserttrueAssertionContextBoundary'lastAssert'BeforeExecutions1BeforeTasks1MaxPictures2AfterActPictures1Interval50改动文件
新增:
packages/core/src/agent/assertion-evidence.ts及单测。只接到现有文件:
types.ts、agent.ts、insight.ts、tasks.ts、task-runner.ts、extraction.ts、report 截图栏、visualizer 回放、@midscene/test选项 schema。未包含:本地 Android
testcases/、生成的 dump、快照换行噪音、二进制、dist。测试计划
packages/core证据链单测(选项归一、历史窗口、最新图优先、MaxPictures = 0、当前截图回退、四段 thought)task-runner只在 Action Space 录before-calling和编号after-calling-Ninsight-extract-prompt发送证据图且不带当前截图tasks-null-data在默认deepAssert: true下 Assert 路径仍可用deepAssertEvidence/buildDeepAssertScreenshotsaiAct再aiAssert,检查 HTML 报告截图栏)